Papers with end-to-end speech translation
JoeyS2T: Minimalistic Speech-to-Text Modeling with JoeyNMT (2022.emnlp-demos)
Copied to clipboard
| Challenge: | JoeyS2T is a simple, lightweight NMT extension for speech-to-text tasks such as automatic speech recognition and end-to end speech translation. |
| Approach: | JoeyS2T is a JoeyNMT extension for automatic speech recognition and end-to-end speech translation. |
| Outcome: | JoeyS2T performs competitively on English speech recognition and English-to-German speech translation benchmarks. |
NeurST: Neural Speech Translation Toolkit (2021.acl-demo)
Copied to clipboard
| Challenge: | a toolkit for speech translation is available for free and provides step-by-step recipes for feature extraction, data preprocessing, distributed training, and evaluation. |
| Approach: | They propose to use NeurST to facilitate speech translation research for NLP researchers . they show experimental results for different benchmark datasets which can be regarded as reliable baselines . |
| Outcome: | The proposed framework provides reliable benchmarks for speech translation research. |
Non-Parametric Domain Adaptation for End-to-End Speech Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | End-to-end speech translation (E2E-ST) systems have received increasing attention due to its less error propagation, lower latency and fewer parameters. |
| Approach: | They propose a non-parametric method that leverages in-domain text translation corpus to achieve domain adaptation for E2E-ST systems. |
| Outcome: | The proposed method outperforms the existing in-domain fine-tuning strategies on the Europarl-ST benchmark. |
Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation (2023.acl-short)
Copied to clipboard
| Challenge: | End-to-end speech translation models have limited training data and are often inefficient due to the inconsistency of length and representation between speech and text. |
| Approach: | They find that the "modality gap" between speech and text data is not a major problem in E2E ST . they decouple the encoder to speech encoder and text encoder, and they find that there is a 'capacity gap' |
| Outcome: | The proposed model achieves 29.0 for en-de and 40.3 for fr on the MuST-C dataset. |
Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | End-to-end speech translation models can be trained to leverage source text . however, since the input modalities are different, it is difficult to leverage the source text successfully. |
| Approach: | They propose to leverage source transcriptions via pre-training and joint training with ASR and NMT tasks. |
| Outcome: | The proposed model predicts paraphrased transcriptions as an auxiliary task with a single decoder. |
Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Developing techniques to support end-to-end speech translation is non-trivial because of the speech-text modality gap. |
| Approach: | They propose a coarse labeling approach that merges vocabulary labels via simple heuristic rules . they propose to use 256-bit truncation, division or modulo operations to regularize the encoder . |
| Outcome: | The proposed method can increase training efficiency while delivering better performance. |
Learning Shared Semantic Space for Speech-to-Text Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | End-to-end speech translation (ST) has been treated as an independent task . however, the modality gap has rendered MT data and its end-to end models incompatible with their ST counterparts. |
| Approach: | They propose to bridge the representation gap between text and audio inputs by projecting audio and text features to a common semantic representation. |
| Outcome: | The proposed model improves performance on two ST benchmarks and achieves 27.1 BLEU on MuST-C EN-DE. |
FCGCL: Fine- and Coarse-Granularity Contrastive Learning for Speech Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to perform implicit knowledge transfer from machine translation to ST model are difficult because of the task complexity and data scarcity. |
| Approach: | They recommend a method which conducts explicit knowledge transfer from MT to ST model by fine and coarse granularity contrastive learning. |
| Outcome: | The proposed method improves the performance of the end-to-end speech translation model on all 8 languages. |
AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | End-to-end speech translation models learn acoustic representations from the encoder, which is not desirable for cross-modal and cross-lingual translation. |
| Approach: | They propose an adaptive speech-to-text translation model that dynamically adapts acoustic states in the decoder. |
| Outcome: | The proposed model outperforms state-of-the-art speech translation models on two widely-used datasets. |
Mutual-Learning Improves End-to-End Speech Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to end-to-end speech translation (E2E) models only allow one way knowledge transfer, which is limited by the performance of the teacher model. |
| Approach: | They propose a one-way knowledge transfer paradigm where the MT and ST models are collaboratively trained and considered as peers rather than teacher/student. |
| Outcome: | The proposed model improves the performance of end-to-end speech translation (ST) task by combining knowledge from two models with peer models. |
LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | ***LLaST*** is a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Approach: | They propose a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Outcome: | The proposed model outperforms the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. |
Rethinking and Improving Multi-task Learning for End-to-end Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | auxiliary tasks are highly consistent with end-to-end speech translation (ST) but their effectiveness has not been thoroughly studied. |
| Approach: | They propose an improved multi-task learning approach for the ST task that bridges the modal gap by mitigating the difference in length and representation. |
| Outcome: | The proposed approach achieves state-of-the-art on the MuST-C dataset with 20.8% of training time required by the current SOTA method. |
CTC-based Non-autoregressive Speech Translation (2023.acl-long)
Copied to clipboard
Chen Xu, Xiaoqian Liu, Xiaowen Liu, Qingxuan Sun, Yuhao Zhang, Murun Yang, Qianqian Dong, Tom Ko, Mingxuan Wang, Tong Xiao, Anxiang Ma, Jingbo Zhu
| Challenge: | End-to-end speech translation (E2E ST) and non-autoregressive (NAR) generation are promising in language and speech processing for their advantages of less error propagation and low latency. |
| Approach: | They develop a model that uses connectionist temporal classification to predict the source and target texts. |
| Outcome: | The proposed model achieves an average BLEU score of 29.5 with a speed-up of 5.67. |
Understanding and Bridging the Modality Gap for Speech Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve end-to-end speech translation (ST) use multitask learning, but there is always a modality gap between ST and MT due to the differences between speech and text. |
| Approach: | They propose a method to bridge the modality gap between ST and MT by leveraging (text) machine translation data. |
| Outcome: | The proposed method bridges the modality gap and achieves significant improvements over baseline in all eight directions. |
Parameter-Efficient Transfer Learning for End-to-end Speech Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to improve end-to-end speech translation are limited by the availability of labeled data. |
| Approach: | They propose a method which utilizes two lightweight adaptation techniques to modulate Attention and the Feed-Forward Network while preserving the capabilities of pre-trained models. |
| Outcome: | The proposed method outperforms baseline models and significantly improves performance in low-resource settings. |